The present invention relates to methods and apparatus for determining deformation of surfaces. More specifically, the present invention relates to methods and apparatus for determining surface deformations based upon incomplete data. Other aspects of the present invention relate to methods and apparatus for animating surfaces based upon surface deformation data.
Some embodiments of the present invention relate to capturing surface deformations of three-dimensional objects (e.g. humans, animals, objects or the like). The surface of a human body moves in many subtle but visually significant ways when performing dynamic activities such as: bending, bulging, jiggling, stretching, and the like. Various embodiments of the present invention are described below with respect to examples of human bodies, however, such examples should not be considered limiting.
Optical motion capture (mocap) techniques have been used to specify motion for cartoon characters, computer-generated characters, and the like for television programs, short and feature-length movies and video games, and the like. Some in the industry refer to the use of mocap techniques as the basis for computer generated imagery as “digital puppeteering.”
The current state of the art for whole body motion capture typically rely upon tracking positions of a set of 40-60 markers placed on a (human) subject. Using such techniques, the motion of the human is reduced to motion of 15-22 rigid body segments. To the extent possible, the markers are placed on joint axes and bony landmarks so that they can be more easily used to approximate the motion of the human skeleton. The range of motion of the human is further reduced by biomechanical invariants, e.g. elbows not bending backwards.
Drawbacks to traditional mocap techniques include that the captured motion of the object often lacks detailed motions that gives the object “life.” In other words, traditional mocap techniques typically record very simplified and gross motions of objects, such as only the motion of a skeleton. Traditional mocap techniques often lack subtle dynamic motion effects of the skin and muscles. Therefore, traditional mocap techniques often fail to capture the correct shapes of bulging muscles, dynamic jiggling of flesh, moving of skin, and the like.
In the current state of the art, based upon the motion captured data, the motion of computer generated humans, for example, are determined. As an example, based upon the motion captured position of the human skeleton at a certain time instance, e.g. a specific frame, a computer model of the human may be posed in the same position.
Subsequently, a variety of techniques are being used to put flesh and skin on top of the skeleton. In the industry this step is often termed “skinning.” In the following discussion, facial movement and hand movement are considered separately, since simplified techniques are often used for those body parts.
One of the common skinning techniques required the CGI modeler to specify a contribution of each bone to the position of the vertices of a skin surface by painting weights on the model. In some cases, the closer a particular skin patch is to a specific joint, the greater the contribution the joint movement is to the position of the skin patch. These techniques, which have been variously called skeleton subspace deformation, single weight enveloping, or skinning, are considered easy to implement and quick to compute.
Drawbacks to such skinning techniques include that with basic implementations, no set of weights will prevent collapsing joints or a “candy wrapper” effect, as illustrated in FIG. 1. This candy wrapper effect occurs because the volume of the body is not considered and preserved when the human bends an arm, for example.
A number of recent techniques have attempted to fix the candy wrapper problems automatically, without significant computational cost. Such techniques have included interpolating spherical rotations rather than performing linear interpolations, approximating the model by swept ellipsoids, adding deformable chunks under the surface of the skin to provide a very simple model of muscle and fat, constructing a simple anatomical model from the outside in, or the like.
Drawbacks to such techniques include that although the candy wrapper effect may be reduced, such techniques do not address the issue of dynamic effects of the skin, such as the above mentioned jiggling of the flesh, muscles bulging due to exertion, and the like.
Other techniques for determining the shape of skin have included having the shape of skin being a function of the pose of the character. Such techniques have included the use of hints based upon key poses of a subject. Some examples have included these techniques along with skeleton subspace deformation, some have included these techniques with radial basis functions to interpolate among a set of example poses of a person, some have included skinning with additional degrees of freedom (and therefore more weights), and/or have included these techniques with additional joints rather than as skinning weights.
Drawbacks to such techniques include that these techniques rely upon specifically posed positions of the character model. Additional drawbacks include that the poses of these models are created by hand by an object modeler. These techniques attempt to model changes in the shape of the body as a function of pose from a modeling stand point, and not upon motion captured data. As a result, such techniques do not model dynamic effects such as changes in shape as a function of the torque being applied at a joint. Further, these techniques thus do not address the issue of dynamic effects of the skin, such as the above mentioned jiggling of the flesh, muscles bulging due to exertion, and the like.
Other data-driven approaches have relied on scanning (e.g. laser scanning) of subjects (e.g. humans) obtained while the character holds specific poses. An example technique correlates a single set of scanned data for an upper body of a character to specific motion captured poses for the character. Superficially, such a technique may appear to have divided movement of a surface into a rigid body (e.g. motion captured pose) and a residual deformation (e.g. a single set of scanned data). However, the movement of the rigid body and the residual deformation, are not based upon motion captured data at a particular time (e.g. a particular frame). As noted above, these data sets are captured at different frames.
Drawbacks to such approaches also include that they do not use dynamic effects in the surface of the skin of the character and the rigid motion of the character at the same period of time (e.g. frame time). Accordingly, for identical poses of the character, the shape of the skin will be the same. In other words, the skin of the character (e.g. pectoral muscles) will have the same shape if the human subject is pointing their toes and an inch off the ground. More specifically, there is no distinction if the human subject is in the process of jumping up or about to land on the ground.
Another technique has been to capture deformations of silhouettes of an object along with motion of a skeleton. Such a system, however is limiting as it merely captures motion along a profile of the object, and not motions of other portions of the object.
Additional approaches have included attempting to realistically model and simulate the underlying musculature of the human body. Such approaches attempt to model complex anatomy of an object, e.g. a human, and also attempt to simulate functionality of the anatomy. For instance, some systems attempt to model the flex of muscles as they apply joint torques. Other systems attempt to model the quasi-static movement of flesh with a finite number of elements, and to perform a finite element analysis. Still other systems use simple dynamic models of the anatomy to add dynamic effects to the motion of skin.
Drawbacks with such approaches include the difficulty in modeling the anatomy, the computational burden of determining the motion of underlying anatomy, the difficulty in controlling and predicting the results of the computations, and the like. Another drawback includes that the result of the simulation system determines the muscle shape, but not how the skin is deformed in response to the muscle shape.
The hands and face are considered special cases for the problem of animating skin motion. Specifically, because positions of faces of characters are relatively planar, conventional motion capture techniques can used more reliably than for other portions of a human. Because facial deformations do not include significant occlusions, they can be captured using capture techniques such as structured light. Additionally, for faces, rigid body motion considerations are typically not considered. In the case of hands, because hands are relatively bony in structure, hand deformations are not well captured using markers.
In light of the above, what is desired are methods and apparatus for capturing and animating dynamic surface deformations of objects without the drawbacks described above.